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Introduction 

In 2004, the National Center for Engineering and Technology Education 
(NCETE) secured funding from the National Science Foundation (NSF) to 
infuse engineering design into the schools through technology education. In 
order to reach this goal the researchers, in cooperation with NCETE, conducted 
a two phase study to identify outcomes for high-school students studying 
engineering. The first study (referred to as a Phase I) focused on students who 
intended to enter an engineering program after high school, answering the 
question: 

What are the engineering student outcomes that prospective engineering 
students in grades 9 through 12 should know and be able to do and prior to 
entering into a post-secondary engineering program? 

This initial study by Childress & Rhodes (2008) started with preexisting items 
selected from ten sources, including focus groups and national standards 
projects. At the end of the Delphi Round 3, very few of these items had been 
dropped since the consensus on all of them was high. Therefore, the researchers 
decided to have selected engineers categorize the outcome items into groups of 
conceptual likeness and to assign categorical names to the groupings. These 
groups then formed the basis of the instruments for the remaining Delphi rounds. 

This resultant baseline of achievement outcomes for prospective engineering 
students was then used to design a modified Delphi instrument for the second 
study (Phase II) reported herein. This study focused on students who were 
enrolled in technology education for general education puiposes and sought 
input from technology education teachers, teacher educators, and supervisors 
regarding the following question: 

What are those engineering outcomes that should be taught in a high 
school technology education program in which the focus is general 
technological literacy and not pre-engineering? 


Craig Rhodes (rhodesc@ncat.edu) is an Associate Professor and Vincent Childress 
(childres@ncat.edu) is a Professor in the Technology Education Program at North Carolina A&T 
State University, Greensboro 
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Background 

Wicklein’s “Five good reasons for engineering design as the focus for 
technology education” (2006; cf. NAE, 2004; Lewis, 2004) effectively identified 
the importance of this study. In presenting the advantages of infusing 
engineering into technology education, he stated that the optimization, analysis, 
and predictions are among the more important things and those students should 
know about engineering. This study takes the next step by identifying the related 
engineering design concepts to be included in a technology education 
curriculum. 

Among recent efforts to identify engineering outcomes for high school 
students, Dearing and Daugherty (2004) used a modified Delphi technique to 
identify concepts that would prepare secondary students for postsecondary 
engineering, in the context of a course promoting technological literacy. Their 
experts were secondary and postsecondary educators. 

Method 

To develop the Delphi instrument for the present study, the results of Phase 
I were categorized and a ranked list of 40 outcomes, recommended by engineers 
for high-school students who want to pursue postsecondary engineering 
education, was determined. 

Twenty-two technology educators served as Phase II Delphi participants: 7 
high-school teachers, 6 administrators, and 9 teacher educators (Table 1). Each 
was paid a participation stipend. Participants were selected based on the 
researchers’ knowledge of their professional expertise, or in some cases, on the 
recommendation of a person deemed qualified for participation. Potential 

Table 1 


Participant demographics 


Gender 

Female 

Male 

13.6% 

86.4% 

^ o\ 

m i—< 

II II 

Race 

Caucasian 

95.5% 

(n = 21) 

African American 

0% 

(n = 0) 

Hispanic 

4.5% 

(n = 1) 

Age 

Mean 

46.8 


Range 

26 to 62 


Years Teaching Experience 

Mean 

21.95 


Range 

4 to 37 


Years Experience in Current TE Role 

Mean 

13.4 


Range 

4 to 31 
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participants were asked to describe their level of knowledge of engineering and 
its relationship to technology education (Table 2). 

No one participating in the study was an employee of an institution that is a 
full partner in the NCETE. Participant #4 and #22 dropped out at Round 1. All 
participants who completed Round 3 were paid a fee for their full participation. 

Table 2 


Participant Expertise 


ID 

No. 

Yrs. 

Experi¬ 

ence 

Position 

Self-Description of Knowledge 

1 

7 

Classroom 

teacher 

Described STLs and relationship to engineering 

2 

25 

Supervisor 

Strong 

3 

7 

Classroom 

teacher 

Past president of TE related association 

4 

7 

Classroom 

teacher 

BS in Mechanical Engineering 

5 

4 

Classroom 

teacher 

MS in Bioengineering 

6 

14 

Teacher 

educator 

High 

7 

36 

Supervisor 

Taught engineering for 15 years; curriculum writer 

8 

32 

Supervisor 

Chair of a standards team; wrote cross reference of 
standards with PLTW. 

9 

6 

Classroom 

teacher 

Did specific research on it in college; writing 
standards for state 

10 

19.5 

Classroom 

teacher 

Role is to prepare students to major in engineering 

11 

36 

Teacher 

educator 

Consulted with engineering firms regularly, co¬ 
inventor of medical research apparatus, keep in 
close contact with engineering departments 

12 

37 

Teacher 

educator 

Have studied and written about engineering 
concepts 

13 

25 

Teacher 

educator 

Unsure. Called for follow-up. Is not an engineer 
but understands rote of engineering in technology 
education 

14 

29 

Teacher 

educator 

Above average 

15 

18 

Classroom 

teacher 

Teach engineering 

16 

31 

Teacher 

educator 

Know enough to be dangerous Works with College 
of Engineering and has published and consulted on 
engineering related projects. 
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Table 2 (continued) 
Participant Expertise 


17 

15 

Teacher 

educator 

Understand principles of engineering through 
applied teaching and learning activities and 
involvement 

18 

24 

Supervisor 

High 

19 

27 

Supervisor/ 

Classroom 

teacher 

Understands the basic engineering process and 
tools needed to be an engineer like math and 
teamwork skills, knowledge of specific engineering 
fields is very minimal 

20 

30 

Teacher 

educator 

On a scale of 1 to 5, 1 am a 4 - better than average, 
not expert 

21 

33 

Teacher 

educator 

Very high 

22 


Supervisor 

(none provided) 


In Rounds 1 through 3, participants were given the list of outcomes items 
from Phase I. They were informed that the items had been selected by engineers 
for inclusion in high school pre-engineering programs. The Phase II participants 
were asked to rate these items, reword them if needed, add and rate new items, 
and provide comments. They were also asked to rank the importance of the items 
using a five point scale (Least Important, Less Important, Important, More 
Important, Most Important; Clark & Wenig, 1999). The descriptions for each of 
these categories are reported in Table 3. The interquartile range (IQR) was used 
as the statistic for variability of rating responses; an IQR of 1 was determined by 
the researchers to indicate consensus on an item (Wicklein, 1993). 

Table 3 

Explanation of ratings 

Rating Statements _ 

1. Least Important: Not necessary for inclusion in a technological literacy 
program. 

2. Less Important: Less than necessary for inclusion in a technological literacy 
program. 

3. Important: Necessary for inclusion in a technological literacy program. 

4. More Important: Essential for inclusion in a technological literacy program. 

5. Most Important: Most essential for inclusion in a technological literacy 

program._ 


Thirty-five of the 44 items from Phase I achieved consensus after Round 3. 
Three additional items were identified by participants after Round 1 and these 
items achieved consensus after Round 2. After Round 3, consensus had been 
reached on 38 engineering outcome items. Eleven items were rated as 
“Important” to include in a technological literacy program, 22 items were rated 
“More Important” and five items were rated “Most Important.” For comparison, 
after three rounds the participants in Phase I reached consensus on what should 
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be taught 1 st , 3 rd , and 7 th places in order of importance. However, in Phase II 
after three rounds, the participants reached consensus on the order of only one 
grouping, Emerging Fields of Engineering, and it was ranked lowest in 
importance. In Phase II, no engineering outcomes items were dropped due to a 
low median importance ratings; however, those outcome items that were not 

Table 4 


Third round consensus ratings of engineering outcomes and rankings of the 
seven outcome groupings _ 


Item 

Rating 

Rank 

Groun: Enaineerina Desian 

Regarding engineering outcomes for Engineering Design the 
student in grades 9-12: 

4 

Rank 

Undeter¬ 

mined 

Is aware of how engineering principles must be applied when 
designing engineering solutions to problems. 

4 

Understands that creativity is an important characteristic for engineers 
to apply in design. 

4 

Believes in his/her ability to design a solution to a problem. 

5 

Recognizes that there are many approaches to design and not just one 
“design process.” 

3 

Understands engineering as it is actually practiced as a future career 
option. 


Groun: Annlication of Enaineerina Desian 

Regarding engineering outcomes for Application of Engineering. 
Design the student in grades 9-12: 

4 

Rank 

Undeter¬ 

mined 

Designs, produces, and tests prototypes of products. 

4 

Understands that there is no perfect design. Designs that are best in 
one respect may be inferior in other ways (cost or appearance). 

Usually some features must be sacrificed as trade-offs to gain other 
features. 

3.5 

Conducts reverse engineering and can analyze how a product or 
process was designed and created. 

4 * 

Understands how to work well on multidisciplinary teams. 

4 

Applies research and development and experimentation in the 
production of new or improved products, processes, and materials. 


Groun: Enaineerina Analysis 

Regarding engineering outcomes for Engineering Analysis the 
student in grades 9-12: 

4 

Rank 

Undeter¬ 

mined 

Uses models to study processes that cannot be studied directly. 

4 

Applies mathematics and science to the engineering process. 

4 

Understands that knowledge of science and mathematics is critical to 
engineering. 

3 

Uses a physical or mathematical model to estimate the probability of 
events. 

3 

Uses optimization techniques to determine optimum solutions to 
problems. 
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Table 4 (continued) 

Third round consensus ratings of engineering outcomes and rankings of the 
seven outcome groupings _ 



Group: Enaineerina and Human Values 

Regarding engineering outcomes for Engin. and Human Values 
the student in grades 9-12: 

4 

Rank 

undeter¬ 

mined 

Practices engineering ethics. 

5 

Is aware of how societal interests, economics, ergonomics, and 
environmental considerations influence a solution. 

5 

Understands how other factors, such as cost, safety, appearance, 
environmental impact, and what will happen if the solution fails must 
be considered when designing engineering solutions to problems. 

4 

Takes human values and limitations into account when designing and 
solving problems. 

5 

Understands that the solution to one problem may create other 
problems. 


Group: Enaineerina Communication 

Regarding engineering outcomes for Engin. Communication the 
student in grades 9-12: 

4 

Rank 

Undeter¬ 

mined 

Understands basic personal computer operations and uses basic 
computer applications such as word processors, spreadsheets, and 
presentation software. 

4 

Provides basic technical presentations, graphics, and reports, and 
communicates verbally information related to engineering processes. 

4 

Uses technical drawings to construct or implement an object, 
structure, or process. 

4 

Visualizes in three dimensions. 

3 

Understands computer-aided engineering. 

3 

Applies the rules of dimensioning and tolerancing. 

4 

Uses computer-aided design to construct technical drawings. 


Group: Enaineerina Science 

Regarding engineering outcomes for Engineering Science the 
student in grades 9- 12: 

5 

Rank 

Undeter¬ 

mined 

Develops basic ability to use, manage, and assess technology. 

3 

Applies knowledge of basic ergonomics to the engineering process. 

4 

Develops basic skill in the use of tools for material processes. 

4 

Applies basic power and energy concepts. 

3.5 

Applies knowledge of the processes for manufacturing products to the 
engineering process. 

4 

Applies knowledge of material processes to the engineering process. 

4 

Applies knowledge of basic mechanics to the engineering process. 

3 

Applies knowledge of basic dynamics and motion of rigid bodies and 
particles to the engineering process. 

4* 

Understands open and closed loop systems. 

3* 

Describes the sources, basic chemical structure, recycling potential, 
and environment al impacts of widely used industrial materials. 
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Table 4 (continued) 

Third round consensus ratings of engineering outcomes and rankings of the 
seven outcome groupings _ 



Group: Emeraina Fields of Enaineerina 

Regarding engineering outcomes for Emerging Fields of Engin. 
the student in grades 9-12: 

3 

Rank 

'yth 

Understands the importance of nanotechnologies in developing the 
next generation of innovations (less power, smaller). 


* Indicates a consensus item added by the participants themselves. 


included in the final list were those for which consensus was not reached. The 
final engineering outcome item ratings and group rankings for Phase II are 
presented in Table 4. A complete statistical analysis of all data, (including non¬ 
consensus items) is available at http://www.ncete.org/flash/Outcomes.pdf. 

As indicated in Table 5, the consensus rate for technology educators in 
Phase II was very similar to the consensus rate for the engineers in Phase I. 

Table 5 


Comparison ofpercentage of consensus outcome items per round per phase 


Round 

Tech Educators (Phase II) 

Engineers (Phase I) 

1 

41% 

42% 

2 

62% 

63% 

3 

80% 

78% 


With the exception of the Emerging Fields of Engineering group, it was 
difficult for both groups of participants (engineers and technology educators) to 
reach consensus on outcome grouping. Among the plausible reasons are that 
both phases dedicated only three rounds to consensus building for the 
groupings, and the initial groupings were juried instead of being crafted by a 
complete Delphi process. 


Discussion 

As of Round 3, there are several consensus items that provide reinforcement 
of the importance of the engineering design processes that the NCETE has 
selected as its professional development focuses: constraints, optimization, 
prediction, and analysis (COPA). Other technology educators may be interested 
in these outcomes also. Those items are presented in Table 6. 
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Table 6 

Consensus items related to constraints, optimization, prediction, and analysis 




Round 3 

Round 2 

Round 1 

Item 

Outcome 

IQR 

Mdn 

— 

l/l 

IQR 

Mdn 

— 

l/l 

IQR 

Mdn 

— 

(fl 

14* 

Uses models to 
study 

processes that 
cannot be 
studied 
directly. 

0* 

4 

.49 

.75* 

4 

.92 

1* 

4 

1.05 

15* 

Uses 

optimization 
techniques to 
determine 
optimum 
solutions to 
problems. 

1* 

3 

.51 

1* 

3 

.74 

1* 

3 

.98 

16* 

Applies 
mathematics 
and science to 
the engineering 
process. 

.75* 

4 

1.02 

1.75 

4 

1.09 

1.25 

4 

1.01 

17* 

Uses a physical 
or 

mathematical 
model to 
estimate the 
probability of 
events. 

1* 

3 

.85 

1.75 

3 

1.05 

1.25 

3 

1.09 

19* 

Understands 
that knowledge 
of science and 
mathematics is 
critical to 
engineering. 

1* 

4 

.95 

1* 

4 

.95 

1* 

4 

1.00 


The use of models for indirect study such as analysis or prediction (Item 
14) had an IQR of zero. Typical comments in support of the More Important 
rating include the following: 

• I believe the ability to use models—mathematical, physical, and virtual, 
is one of the most important skills we can teach in technology education. 

• Modeling is part of the 3-12 STL Standards and should be central to 
what we teach since modeling is such a powerful and universal tool. 

• Making decisions based on models before construction is valuable. 
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Optimization (Item 15) was rated as Important to include in a technological 
literacy program. Comments related to optimization include the following: 

• Calculus is not needed for problems solved related to a technology 
education course. 

• For technological literacy, applying math and science to a design process 
would be vital for optimization and adhering to constraints, as well as 
analyzing data during testing. 

• This is what engineers do. Don’t bother to teach “engineering” if you 
take the math and science out. There is plenty that can be done with 
high-school appropriate mathematics. 

Mathematical analysis-related Items 16 and 19 and predication-related Item 
17 were rated More Important and Important respectively to include in a 
technological literacy program. Supporting comments related to the use of 
mathematics and science for the analysis items included the following: 

• STL currently calls for recognition of such connections between science 
and mathematics to all technological processes. 

• The way this is stated, it seems more like a value or feeling than a 
technical concept or skill and therefore not that difficult to comprehend. 
It does not state that one know how to apply science and mathematics to 
the study of engineering which in my opinion would warrant greater 
importance. 

• Learn by doing 

Comments supporting the prediction outcome included the following: 

• While the background needed to develop mathematical models on their 
own may be lacking, students need to be able to work with these models 
in a meaningful way—not just plugging in data, but interpreting formulas 
and results, optimizing, etc. “Probability” is really the wrong term, 
though. 

• Virtual modeling is critical to understanding the engineering process 
because it allows for iterative processing without cost of trials. Perfect 
for tech ed because you get to simulate what engineers do with limited 
resources. 

• We in TE need to step up and model our solutions and outcomes more 
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Table 7 

Consensus items related to constraints 


Item 

Outcome 

Round 3 

Round 2 

Round 1 

IQR 

Mdn 

— 

fl 

IQR 

Mdn 

— 

(fl 

IQR 

Mdn 

— 

(fl 

21 

* 

Is aware of 

how societal 

interests, 

economics, 

...environment 

al.. .influence a 

solution. 

1* 

5 

.604 

1* 

5 

.606 

1* 

5 

.790 

22 

* 

Understands 
how other 
factors, such as 
cost...must be 
considered 
when 

designing... 

1* 

5 

.995 

1* 

5 

1.08 

1* 

5 

1.07 


Constraints (Items 21 and 22) were rated as Most Important to include in a 
technological literacy program. See Table 7. Comments in support of the 
importance of constraints were. 

• Considers when designing should be used 

• This is why technological literacy (an outcome we should hope to attain 
in all students, not just those seeking engineering careers in the future) is 
so important 

• Definite connection to real life: “each action or decision could affect 
others.” 

Technology educators, interestingly, rated approaches to the design 
process differently than did engineers . The technology educators also rated 
technical drawing slightly higher than did the engineers. After Round 3 for this 
grouping, the engineers ended with an IQR of 1 while the technology educators 
has an IQR of zero, as reported in Table 8. 
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Table 8 


Consensus items related to the design process 


Item 

Outcome 

Round 3 

Round 2 

Round 1 

IQR 

Mdn 

— 

l/l 

IQR 

Mdn 

— 

l/l 

IQR 

Mdn 

— 

l/l 

6* 

Recognizes 
that there are 
many 

approaches to 
design and not 
just one 
“design 
process.” 

1* 

5 

.605 

1* 

5 

.605 

1* 

5 

1.06 

28* 

Uses technical 
drawings to 
construct or 
implement an 
object, 
structure, or 
process. 

0* 

4 

.459 

0* 

4 

.887 

.5* 

4 

.669 


Supporting comments related to recognizing that there is a variety of design 
processes included: 

• General technological literacy ability. 

• It is important, but learning the design process as something that is 
discursive would be fundamental, and perhaps learners who can be 
identified as more creative would be encouraged to take multiple 
approaches to design... 

• Multiple solutions approach to learning is a characteristic of a 
technologically literate person 

Comments related to support of the technical drawing outcome included: 

• Without technical drawings to plan the construction, students are just 
creating art, no? How do you replicate a design if needed? 

• Free hand sketching is needed and can be taught. No time for CAD class 
prior to TE class. Good class if you go to college for Engineering] 
Vocational[y]. 

• I will move to the majority here because your designs and ideas are only 
as good as they can be expressed to another who might have to use your 
graphic representation of them in your absence. 

These findings were similar to those of Dearing and Daugherty in their modified 
Delphi study. 
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Technology educators added an item that did not appear on their Round 1 
instrument because it was not a consensus item for the engineers. However, it did 
appear on the Round 1 instrument for the engineers. This item relates to the 
NAE’s prediction that working on teams will be very important to engineering in 
the future (Item 47 below in Table 9). 

Table 9 


Consensus item related to design teams 


Item 

Outcome 

Round 3 

Round 2 

IQR 

Mdn 

Q 

fi 

IQR 

Mdn 

— 

fl 

Added 47* 
add to Grp 2 

Understands how to work 
well on multidisciplinary 
teams. 

1* 

4 

.67 

1* 

4 

1.07 


Comments related to support of the multidisciplinary teams outcome 
included: 

• This is a critical skill addressed by ABET that is applicable to all 
students in technology education. 

• Understands how to work well on multidisciplinary design teams. 

• Important addition 

Technology education participants did not reach consensus on the 
importance of managing the engineering design process. Related comments are 
included following Table 10 below. 

Table 10 


Statistics for non-consensus item related to managing the engineering design 
process _ 


Item 

Outcome 

Round 3 

Round 2 

Round 1 

IQR 

Mdn 

— 

(fl 

IQR 

Mdn 

Q 

t n 

IQR 

Mdn 

— 

(fl 

8 

Organizes and 
manages the 
engineering, 
design process 
that includes 
optimal 
use... materials 
. processes... 

2 

4 

1.02 

2 

3.5 

1.03 

2 

4 

1.12 


• Optimizing is covered below. I do not feel that “engineering 

management and organization” as a process should play a significant role 
in technological literacy. 


- 80 - 








Journal of Technology Education 


Vol. 21 No. 2, Spring 2010 


• Important for engineering literacy but not essential for technological 
literacy. I would prefer the word “engineering” be deleted from this 
learning outcome statement. 

• Important for engineering literacy, but less important for technological 
literacy. 

Dealing and Daugherty (2004) described a modified Delphi study that they 
conducted with technology teachers, technology teacher educators, and 
engineering educators. The puipose of the study was to identify those curricular 
concepts that are necessary to teach high school students in order to prepare 
them for postsecondary engineering education, while preserving the mission of 
teaching technological literacy. Dealing and Daugherty developed a 
predetermined list based on information from Project Lead the Way, CORD’s 
Principles of Technology, The Standards for Technological Literacy, ASEE, and 
others. Participants were to decide if a concept should be included in such a 
curriculum. 

Dearing and Daugherty measured consensus in terms of an item’s standard 
deviation. Fifty-two concepts on their list met the criterion for consensus and 
were retained. While the purposes of the Dearing and Daugherty study differed 
from this study, they were both focused on technological literacy. Twenty-nine 
of the engineering consensus items in the Dearing and Daugherty study overlap 
conceptually with the consensus items in this study. 

Finally, it should be noted that the Corporate Member Council of the 
American Society for Engineering Education has been working on a set of 
“...National Content Standards for K-12 Engineering/Engineering 
Technology...” (Morrison, 2007, p. 1). Most of the 70 outcomes derived from 
these standards overlap with the findings of this study. However, it should be 
noted that the purposes of the Corporate Member Council’s study differ from the 
purpose of this study. The final findings of the Corporate Member Council’s 
study were not available at the time this article was written. 

Recommendations 

The following recommendations are offered to technology education teacher 
educators, technology education teachers, and technology education 
administrators. 

1. Conduct professional development in which teachers are provided the 
opportunity and guidance to infiise those engineering outcomes agreed 
upon by the participants into the teachers’ own technological literacy 
curricula. 

2. Enhance technology education by infusing selected engineering 
outcomes into the technology education curriculum for non-pre 
engineering curricula. 
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Implications for Technology Education Curriculum and Instruction 

There are at least two primary points of view regarding the application of 
the items identified in this study to general technological literacy programs— 
those that are not pre-engineering. One is that a curriculum could be designed 
that is heavily influenced by engineering contexts. Those outcome items 
identified herein that overlap with the Standards for Technological Literacy 
would address the same standard, but the delivery would be set in the context of 
engineering. A second point of view is that the outcomes identified herein would 
be used sparingly in a program intended to develop technological literacy; 
engineering would be one of many contexts and topics of study included. 
Although agreement was reached on only a few groupings, they may still be of 
use to teachers insofar as the engineers agreed that Group 1, Engineering Design, 
was the group of outcome items of primary importance in a curriculum that is 
crowded and has limited time to dedicate to engineering outcomes. Likewise, 
Group 7, Emerging Fields of Engineering, represents the least important 
grouping that one would want to teach if time were limited. Finally, a crowded 
curriculum will actually provide less opportunity for students to learn about 
engineering in meaningful ways—meaningful ways that will tend to attract 
underrepresented populations to STEM areas. Just because consensus was 
reached on as many as 38 different outcome items in this study does not mean 
that all of them should be taught in one course. They should be applied as 
needed and when pertinent over the span of the ninth through twelfth grades. 
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